The Convolution Operation

\[S(i, j) = (I * K)(i, j) = \sum_{m} \sum_{n} I(i+m, j+n) K(m, n)\]

Where: * \((i, j)\) are the pixel coordinates. * \((m, n)\) are the kernel offsets.

Pooling Layers

Pooling provides invariance to small translations and reduces dimensionality (downsampling).

Max Pooling

Selects the maximum activation in the receptive field: \[y_{i,j,k} = \max_{(p,q) \in \mathcal{R}_{i,j}} x_{p,q,k}\]

Average Pooling

Calculates the arithmetic mean. Generally, Max Pooling performs better for identifying dominant features (edges, textures).

Activation Functions

Linear convolution is insufficient for approximating non-linear functions.

ReLU (Rectified Linear Unit): \[f(x) = \max(0, x)\]

• Sparsity: Activations \(< 0\) are zeroed out.
• Gradient Propagation: Mitigates vanishing gradient problem compared to Sigmoid/Tanh.

Variants: Leaky ReLU, ELU, GELU (Gaussian Error Linear Unit).

Backpropagation in CNNs

Training requires computing gradients w.r.t weights \(W\) using the Chain Rule.

For a convolution layer \(l\): \[\frac{\partial L}{\partial W^{(l)}} = \frac{\partial L}{\partial \text{out}^{(l)}} * \text{in}^{(l)}\]

Where the gradient is computed via convolution between the incoming error signal and the input activations from the previous layer.

Implementation: PyTorch Snippet

import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        # Feature Extraction
        self.features = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        # Classifier
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(64 * 7 * 7, 128), # Assuming 28x28 input
            nn.ReLU(),
            nn.Linear(128, 10)
        )

    def forward(self, x):
        x = self.features(x)
        x = self.classifier(x)
        return x